simple-zstd 2.0.0-0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/ci.yml +46 -0
- package/.github/workflows/release.yml +45 -0
- package/.prettierignore +5 -0
- package/.prettierrc +8 -0
- package/.release-it.json +28 -0
- package/CHANGELOG.md +26 -0
- package/README.md +657 -110
- package/dist/src/buffer-writable.d.ts +12 -0
- package/dist/src/buffer-writable.js +41 -0
- package/dist/src/index.d.ts +49 -0
- package/dist/src/index.js +430 -0
- package/dist/src/peek-transform.d.ts +16 -0
- package/dist/src/peek-transform.js +145 -0
- package/dist/src/process-duplex.d.ts +11 -0
- package/dist/src/process-duplex.js +157 -0
- package/dist/src/process-queue.d.ts +8 -0
- package/dist/src/process-queue.js +94 -0
- package/dist/src/types.d.ts +34 -0
- package/dist/src/types.js +3 -0
- package/eslint.config.js +49 -0
- package/package.json +29 -21
- package/src/buffer-writable.ts +30 -0
- package/src/index.ts +472 -0
- package/src/is-zst.d.ts +5 -0
- package/src/peek-transform.ts +153 -0
- package/src/process-duplex.ts +164 -0
- package/src/process-queue.ts +97 -0
- package/src/types.ts +35 -0
- package/tsconfig.json +110 -0
- package/.eslintrc.js +0 -19
- package/.nyc_output/88144029-aa5a-40fb-a072-f586e59e4a38.json +0 -1
- package/.nyc_output/cb2a47ed-9ada-42b3-863b-b8437ca49715.json +0 -1
- package/.nyc_output/processinfo/88144029-aa5a-40fb-a072-f586e59e4a38.json +0 -1
- package/.nyc_output/processinfo/cb2a47ed-9ada-42b3-863b-b8437ca49715.json +0 -1
- package/.nyc_output/processinfo/index.json +0 -1
- package/.travis.yml +0 -9
- package/buffer-writable.js +0 -23
- package/coverage/base.css +0 -224
- package/coverage/block-navigation.js +0 -87
- package/coverage/buffer-writable.js.html +0 -154
- package/coverage/favicon.png +0 -0
- package/coverage/index.html +0 -146
- package/coverage/index.js.html +0 -910
- package/coverage/oven.js.html +0 -244
- package/coverage/prettify.css +0 -1
- package/coverage/prettify.js +0 -2
- package/coverage/sort-arrow-sprite.png +0 -0
- package/coverage/sorter.js +0 -196
- package/index.js +0 -295
- package/process-queue.js +0 -74
package/README.md
CHANGED
|
@@ -1,183 +1,730 @@
|
|
|
1
1
|
# simple-zstd
|
|
2
2
|
|
|
3
3
|
[](https://travis-ci.org/Stieneee/simple-zstd)
|
|
4
|
-
[](https://packagephobia.now.sh/result?p=simple-zstd)
|
|
5
4
|
[](https://choosealicense.com/licenses/mit/)
|
|
6
5
|
|
|
7
|
-
Node.js interface to system
|
|
6
|
+
Node.js interface to system-installed Zstandard (zstd) with TypeScript support.
|
|
8
7
|
|
|
9
|
-
##
|
|
8
|
+
## Overview
|
|
10
9
|
|
|
11
|
-
|
|
12
|
-
In summary this package, like simple-git, attempts to provide a lightwieght wrapper around the system installed ZSTD binary.
|
|
13
|
-
This provides a more stable package in comparison to a package that builds against a library at the cost of having to manage child process.
|
|
10
|
+
simple-zstd is a lightweight wrapper around the system-installed zstd binary, inspired by simple-git's approach of wrapping system binaries rather than building against native libraries. This provides a more stable and portable solution at the cost of requiring zstd to be installed on the system.
|
|
14
11
|
|
|
15
|
-
|
|
12
|
+
### Features
|
|
16
13
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
14
|
+
- **TypeScript Support**: Full TypeScript definitions and modern ES modules
|
|
15
|
+
- **Multiple Interfaces**: Static functions, buffer methods, and class-based API with process pooling
|
|
16
|
+
- **Promise-Based**: All operations return promises for modern async/await patterns
|
|
17
|
+
- **Stream & Buffer**: Support for both streaming and buffer-based compression/decompression
|
|
18
|
+
- **Smart Decompression**: Automatic detection and passthrough of non-compressed data
|
|
19
|
+
- **Dictionary Support**: Use compression dictionaries via Buffer or file path
|
|
20
|
+
- **Process Pooling**: Pre-spawn child processes for latency-sensitive applications
|
|
21
|
+
- **Node.js 18+**: Built on modern Node.js features
|
|
21
22
|
|
|
22
|
-
|
|
23
|
-
This promise will return the stream or a buffer depending on the function type called.
|
|
23
|
+
## Requirements
|
|
24
24
|
|
|
25
|
-
|
|
26
|
-
|
|
25
|
+
- **Node.js**: >= 18.0.0
|
|
26
|
+
- **zstd**: Must be installed and available on system PATH
|
|
27
27
|
|
|
28
|
-
##
|
|
28
|
+
## Installation
|
|
29
29
|
|
|
30
|
-
|
|
31
|
-
It will work in both Linux and Windows environments assuming the zstd(.exe) can be found on the path.
|
|
30
|
+
### Install zstd
|
|
32
31
|
|
|
33
|
-
|
|
32
|
+
**Ubuntu/Debian:**
|
|
34
33
|
|
|
35
|
-
|
|
34
|
+
```bash
|
|
35
|
+
sudo apt install zstd
|
|
36
|
+
```
|
|
36
37
|
|
|
37
|
-
|
|
38
|
+
**macOS:**
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
brew install zstd
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
**Windows:**
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
choco install zstd
|
|
48
|
+
# or download from: https://github.com/facebook/zstd/releases
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
### Install simple-zstd
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
npm install simple-zstd
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## API Reference
|
|
58
|
+
|
|
59
|
+
### Types
|
|
60
|
+
|
|
61
|
+
```typescript
|
|
62
|
+
import { Duplex } from "node:stream";
|
|
63
|
+
import { SpawnOptions } from "node:child_process";
|
|
64
|
+
import { DuplexOptions } from "node:stream";
|
|
65
|
+
|
|
66
|
+
interface ZSTDOpts {
|
|
67
|
+
dictionary?: Buffer | { path: string }; // Compression dictionary
|
|
68
|
+
zstdOptions?: string[]; // CLI args to pass to zstd (e.g., ['--ultra'])
|
|
69
|
+
spawnOptions?: SpawnOptions; // Node.js child_process spawn options
|
|
70
|
+
streamOptions?: DuplexOptions; // Node.js stream options
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
interface PoolOpts {
|
|
74
|
+
compressQueueSize?: number; // Number of pre-spawned compression processes
|
|
75
|
+
decompressQueueSize?: number; // Number of pre-spawned decompression processes
|
|
76
|
+
compressQueue?: {
|
|
77
|
+
compLevel?: number;
|
|
78
|
+
dictionary?: Buffer | { path: string };
|
|
79
|
+
zstdOptions?: string[];
|
|
80
|
+
spawnOptions?: SpawnOptions;
|
|
81
|
+
streamOptions?: DuplexOptions;
|
|
82
|
+
};
|
|
83
|
+
decompressQueue?: {
|
|
84
|
+
dictionary?: Buffer | { path: string };
|
|
85
|
+
zstdOptions?: string[];
|
|
86
|
+
spawnOptions?: SpawnOptions;
|
|
87
|
+
streamOptions?: DuplexOptions;
|
|
88
|
+
};
|
|
89
|
+
}
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
### Static Functions
|
|
93
|
+
|
|
94
|
+
```typescript
|
|
95
|
+
// Compression
|
|
96
|
+
compress(compLevel: number, opts?: ZSTDOpts): Promise<Duplex>
|
|
97
|
+
compressBuffer(buffer: Buffer, compLevel: number, opts?: ZSTDOpts): Promise<Buffer>
|
|
98
|
+
|
|
99
|
+
// Decompression (with automatic passthrough for non-compressed data)
|
|
100
|
+
decompress(opts?: ZSTDOpts): Promise<Duplex>
|
|
101
|
+
decompressBuffer(buffer: Buffer, opts?: ZSTDOpts): Promise<Buffer>
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### Class: SimpleZSTD
|
|
38
105
|
|
|
39
|
-
`
|
|
106
|
+
The `SimpleZSTD` class provides process pooling for better performance when performing many compression/decompression operations.
|
|
107
|
+
|
|
108
|
+
```typescript
|
|
109
|
+
class SimpleZSTD {
|
|
110
|
+
// Static factory method (recommended)
|
|
111
|
+
static create(poolOptions?: PoolOpts): Promise<SimpleZSTD>;
|
|
112
|
+
|
|
113
|
+
// Instance methods
|
|
114
|
+
compress(compLevel?: number): Promise<Duplex>;
|
|
115
|
+
compressBuffer(buffer: Buffer, compLevel?: number): Promise<Buffer>;
|
|
116
|
+
decompress(): Promise<Duplex>;
|
|
117
|
+
decompressBuffer(buffer: Buffer): Promise<Buffer>;
|
|
118
|
+
destroy(): void;
|
|
119
|
+
|
|
120
|
+
// Statistics
|
|
121
|
+
get queueStats(): {
|
|
122
|
+
compress: { hits: number; misses: number };
|
|
123
|
+
decompress: { hits: number; misses: number };
|
|
124
|
+
};
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**Note:** Use the static `create()` method instead of the constructor. The constructor is private to ensure proper async initialization.
|
|
40
129
|
|
|
41
130
|
## Usage
|
|
42
131
|
|
|
43
|
-
|
|
132
|
+
### Example 1: Stream Interface (TypeScript)
|
|
44
133
|
|
|
45
|
-
```
|
|
46
|
-
|
|
134
|
+
```typescript
|
|
135
|
+
import fs from "node:fs";
|
|
136
|
+
import { pipeline } from "node:stream/promises";
|
|
137
|
+
import { compress, decompress } from "simple-zstd";
|
|
47
138
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
139
|
+
async function copyFile() {
|
|
140
|
+
const c = await compress(3); // Compression level 3
|
|
141
|
+
const d = await decompress();
|
|
142
|
+
|
|
143
|
+
await pipeline(
|
|
144
|
+
fs.createReadStream("example.txt"),
|
|
145
|
+
c,
|
|
146
|
+
d,
|
|
147
|
+
fs.createWriteStream("example_copy.txt")
|
|
148
|
+
);
|
|
53
149
|
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
function decompressBuffer(buffer, spawnOptions, streamOptions, zstdOptions, dictionary) // returns a promise that resolves to a buffer
|
|
150
|
+
console.log("File compressed and decompressed!");
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
copyFile().catch(console.error);
|
|
59
154
|
```
|
|
60
155
|
|
|
61
|
-
|
|
62
|
-
The function names are the same on the class however the options can not be changed from the class constructor.
|
|
156
|
+
### Example 2: Buffer Interface (TypeScript)
|
|
63
157
|
|
|
64
|
-
|
|
158
|
+
```typescript
|
|
159
|
+
import { compressBuffer, decompressBuffer } from "simple-zstd";
|
|
65
160
|
|
|
66
|
-
|
|
67
|
-
|
|
161
|
+
async function processBuffer() {
|
|
162
|
+
const buffer = Buffer.from("this is a test");
|
|
163
|
+
|
|
164
|
+
// Compress with level 3
|
|
165
|
+
const compressed = await compressBuffer(buffer, 3);
|
|
166
|
+
console.log(
|
|
167
|
+
`Original: ${buffer.length} bytes, Compressed: ${compressed.length} bytes`
|
|
168
|
+
);
|
|
169
|
+
|
|
170
|
+
// Decompress
|
|
171
|
+
const decompressed = await decompressBuffer(compressed);
|
|
172
|
+
console.log(decompressed.toString()); // "this is a test"
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
processBuffer().catch(console.error);
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
### Example 3: CommonJS Usage
|
|
68
179
|
|
|
69
180
|
```javascript
|
|
70
|
-
const fs = require(
|
|
71
|
-
const { pipeline } = require(
|
|
72
|
-
const {compress, decompress} = require(
|
|
181
|
+
const fs = require("fs");
|
|
182
|
+
const { pipeline } = require("node:stream/promises");
|
|
183
|
+
const { compress, decompress } = require("simple-zstd");
|
|
73
184
|
|
|
74
185
|
async function copyFile() {
|
|
75
186
|
const c = await compress(3);
|
|
76
187
|
const d = await decompress();
|
|
77
188
|
|
|
78
189
|
await pipeline(
|
|
79
|
-
fs.createReadStream(
|
|
190
|
+
fs.createReadStream("example.txt"),
|
|
80
191
|
c,
|
|
81
192
|
d,
|
|
82
|
-
fs.createWriteStream(
|
|
83
|
-
() => {
|
|
84
|
-
console.log('The file has been compressed and decompressed to example_copy.txt')
|
|
85
|
-
}
|
|
193
|
+
fs.createWriteStream("example_copy.txt")
|
|
86
194
|
);
|
|
195
|
+
|
|
196
|
+
console.log("File compressed and decompressed!");
|
|
87
197
|
}
|
|
88
198
|
|
|
89
|
-
copyFile();
|
|
199
|
+
copyFile().catch(console.error);
|
|
200
|
+
```
|
|
90
201
|
|
|
202
|
+
### Example 4: Class Interface with Process Pooling
|
|
203
|
+
|
|
204
|
+
The `SimpleZSTD` class pre-spawns zstd processes for lower latency. This is ideal for high-throughput scenarios.
|
|
205
|
+
|
|
206
|
+
```typescript
|
|
207
|
+
import fs from "node:fs";
|
|
208
|
+
import { pipeline } from "node:stream/promises";
|
|
209
|
+
import { SimpleZSTD } from "simple-zstd";
|
|
210
|
+
|
|
211
|
+
async function processMultipleFiles() {
|
|
212
|
+
// Create instance with process pools using static factory method
|
|
213
|
+
const zstd = await SimpleZSTD.create({
|
|
214
|
+
compressQueueSize: 2, // Pre-spawn 2 compression processes
|
|
215
|
+
decompressQueueSize: 2, // Pre-spawn 2 decompression processes
|
|
216
|
+
compressQueue: {
|
|
217
|
+
compLevel: 3, // Default compression level for pool
|
|
218
|
+
},
|
|
219
|
+
});
|
|
220
|
+
|
|
221
|
+
try {
|
|
222
|
+
// Process first file with pool default (level 3)
|
|
223
|
+
const c1 = await zstd.compress();
|
|
224
|
+
const d1 = await zstd.decompress();
|
|
225
|
+
|
|
226
|
+
await pipeline(
|
|
227
|
+
fs.createReadStream("file1.txt"),
|
|
228
|
+
c1,
|
|
229
|
+
d1,
|
|
230
|
+
fs.createWriteStream("file1_copy.txt")
|
|
231
|
+
);
|
|
232
|
+
|
|
233
|
+
console.log("File 1 processed!");
|
|
234
|
+
|
|
235
|
+
// Process second file with custom compression level (bypasses pool)
|
|
236
|
+
const c2 = await zstd.compress(19); // Override with level 19
|
|
237
|
+
const d2 = await zstd.decompress();
|
|
238
|
+
|
|
239
|
+
await pipeline(
|
|
240
|
+
fs.createReadStream("file2.txt"),
|
|
241
|
+
c2,
|
|
242
|
+
d2,
|
|
243
|
+
fs.createWriteStream("file2_copy.txt")
|
|
244
|
+
);
|
|
245
|
+
|
|
246
|
+
console.log("File 2 processed!");
|
|
247
|
+
|
|
248
|
+
// Check pool statistics
|
|
249
|
+
console.log("Pool stats:", zstd.queueStats);
|
|
250
|
+
// Example output: { compress: { hits: 1, misses: 1 }, decompress: { hits: 2, misses: 0 } }
|
|
251
|
+
// Note: compress shows 1 miss because we used custom level for file2
|
|
252
|
+
} finally {
|
|
253
|
+
// Clean up all child processes
|
|
254
|
+
zstd.destroy();
|
|
255
|
+
}
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
processMultipleFiles().catch(console.error);
|
|
91
259
|
```
|
|
92
260
|
|
|
93
|
-
### Example
|
|
261
|
+
### Example 5: Using Compression Dictionaries
|
|
94
262
|
|
|
95
|
-
|
|
263
|
+
```typescript
|
|
264
|
+
import fs from "node:fs";
|
|
265
|
+
import { compressBuffer, decompressBuffer, SimpleZSTD } from "simple-zstd";
|
|
96
266
|
|
|
97
|
-
|
|
98
|
-
|
|
267
|
+
// Static functions with dictionaries
|
|
268
|
+
async function useDictionaryStatic() {
|
|
269
|
+
const dictionary = fs.readFileSync("my-dictionary.zstd");
|
|
270
|
+
const data = Buffer.from("Sample text to compress");
|
|
99
271
|
|
|
100
|
-
|
|
101
|
-
const
|
|
272
|
+
// Compress with dictionary
|
|
273
|
+
const compressed = await compressBuffer(data, 3, { dictionary });
|
|
102
274
|
|
|
103
|
-
|
|
104
|
-
const decompressed = await decompressBuffer(compressed);
|
|
275
|
+
// Decompress with same dictionary
|
|
276
|
+
const decompressed = await decompressBuffer(compressed, { dictionary });
|
|
277
|
+
|
|
278
|
+
console.log(decompressed.toString()); // "Sample text to compress"
|
|
279
|
+
}
|
|
105
280
|
|
|
106
|
-
|
|
281
|
+
// Class with dictionaries (supports Buffer or file path)
|
|
282
|
+
async function useDictionaryClass() {
|
|
283
|
+
const dictionary = fs.readFileSync("my-dictionary.zstd");
|
|
284
|
+
|
|
285
|
+
const zstd = await SimpleZSTD.create({
|
|
286
|
+
compressQueue: {
|
|
287
|
+
compLevel: 3,
|
|
288
|
+
dictionary, // Can be Buffer or { path: '/path/to/dict' }
|
|
289
|
+
},
|
|
290
|
+
decompressQueue: {
|
|
291
|
+
dictionary, // Same dictionary for decompression
|
|
292
|
+
},
|
|
293
|
+
});
|
|
294
|
+
|
|
295
|
+
try {
|
|
296
|
+
const data = Buffer.from("Sample text to compress");
|
|
297
|
+
const compressed = await zstd.compressBuffer(data);
|
|
298
|
+
const decompressed = await zstd.decompressBuffer(compressed);
|
|
299
|
+
console.log(decompressed.toString()); // "Sample text to compress"
|
|
300
|
+
} finally {
|
|
301
|
+
zstd.destroy();
|
|
302
|
+
}
|
|
107
303
|
}
|
|
108
304
|
|
|
109
|
-
|
|
305
|
+
useDictionaryStatic().catch(console.error);
|
|
110
306
|
```
|
|
111
307
|
|
|
112
|
-
|
|
308
|
+
**Dictionary Caching:** When using dictionary Buffers with static functions, simple-zstd automatically caches the temporary dictionary files using SHA-256 hashing. This means:
|
|
309
|
+
- ✅ Multiple calls with the **same dictionary Buffer** reuse the same temp file
|
|
310
|
+
- ✅ No performance penalty for repeated operations with dictionaries
|
|
311
|
+
- ✅ Automatic cleanup when the dictionary is no longer in use
|
|
312
|
+
- ✅ **Fixes exponential slowdown** when compressing thousands of items with dictionaries
|
|
113
313
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
Options can not be changed once the class is instantiated.
|
|
117
|
-
This due to the fact that the class will start to spawn child processes with settings passed to the constructor.
|
|
118
|
-
Calling ```destroy()``` will kill all child processes.
|
|
314
|
+
```typescript
|
|
315
|
+
const dict = fs.readFileSync('my-dict.zstd');
|
|
119
316
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
const { SimpleZSTD } = require('simple-zstd');
|
|
124
|
-
|
|
125
|
-
poolOptions = {
|
|
126
|
-
compressQueue: {
|
|
127
|
-
targetSize: 1 // this determines how many instances of zstd are spawned into the qeueue. leaving it empty will result child processes to bring spawned as needed.
|
|
128
|
-
compLevel,
|
|
129
|
-
spawnOptions,
|
|
130
|
-
streamOptions,
|
|
131
|
-
zstdOptions
|
|
132
|
-
},
|
|
133
|
-
decompressQueue: {
|
|
134
|
-
targetSize: 1 // there is a queue for both compression and decompression
|
|
135
|
-
spawnOptions,
|
|
136
|
-
streamOptions,
|
|
137
|
-
zstdOptions
|
|
138
|
-
},
|
|
317
|
+
// These 1000 operations will only create ONE temp file total
|
|
318
|
+
for (let i = 0; i < 1000; i++) {
|
|
319
|
+
await compressBuffer(data[i], 3, { dictionary: dict });
|
|
139
320
|
}
|
|
321
|
+
// Temp file is automatically cleaned up when no longer referenced
|
|
322
|
+
```
|
|
140
323
|
|
|
141
|
-
|
|
142
|
-
const i = new SimpleZSTD(poolOptions, dictionary); // poolOptions set options for the compressions and decompressiong porcess pools. dictionary is optional and set for both pools.
|
|
324
|
+
### Example 6: Smart Decompression (Auto-detect)
|
|
143
325
|
|
|
144
|
-
|
|
145
|
-
const d = await i.decompress();
|
|
326
|
+
The decompression functions automatically detect if data is zstd-compressed and pass through uncompressed data unchanged.
|
|
146
327
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
);
|
|
328
|
+
```typescript
|
|
329
|
+
import { decompressBuffer } from "simple-zstd";
|
|
330
|
+
|
|
331
|
+
async function smartDecompress() {
|
|
332
|
+
const plainText = Buffer.from("not compressed");
|
|
333
|
+
const result = await decompressBuffer(plainText);
|
|
334
|
+
|
|
335
|
+
// Non-compressed data passes through unchanged
|
|
336
|
+
console.log(result.toString()); // "not compressed"
|
|
157
337
|
}
|
|
158
338
|
|
|
159
|
-
|
|
339
|
+
smartDecompress().catch(console.error);
|
|
160
340
|
```
|
|
161
341
|
|
|
162
|
-
##
|
|
342
|
+
## Advanced Options
|
|
163
343
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
344
|
+
### Custom zstd Options
|
|
345
|
+
|
|
346
|
+
Pass any command-line option to the zstd process via `zstdOptions`:
|
|
347
|
+
|
|
348
|
+
```typescript
|
|
349
|
+
import { compress } from "simple-zstd";
|
|
350
|
+
|
|
351
|
+
// Use ultra compression (level 22)
|
|
352
|
+
const stream = await compress(22, {
|
|
353
|
+
zstdOptions: ["--ultra"],
|
|
354
|
+
});
|
|
355
|
+
|
|
356
|
+
// Multiple options
|
|
357
|
+
const stream2 = await compress(19, {
|
|
358
|
+
zstdOptions: ["--ultra", "--long"],
|
|
359
|
+
});
|
|
360
|
+
```
|
|
361
|
+
|
|
362
|
+
### Spawn Options
|
|
363
|
+
|
|
364
|
+
Control the child process spawn behavior:
|
|
365
|
+
|
|
366
|
+
```typescript
|
|
367
|
+
import { compress } from "simple-zstd";
|
|
368
|
+
|
|
369
|
+
const stream = await compress(3, {
|
|
370
|
+
spawnOptions: {
|
|
371
|
+
cwd: "/custom/working/directory",
|
|
372
|
+
env: { ...process.env, CUSTOM_VAR: "value" },
|
|
373
|
+
},
|
|
374
|
+
});
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
### Stream Options
|
|
378
|
+
|
|
379
|
+
Customize the Duplex stream behavior:
|
|
380
|
+
|
|
381
|
+
```typescript
|
|
382
|
+
import { compress } from "simple-zstd";
|
|
383
|
+
|
|
384
|
+
const stream = await compress(3, {
|
|
385
|
+
streamOptions: {
|
|
386
|
+
highWaterMark: 64 * 1024, // 64KB buffer
|
|
387
|
+
},
|
|
388
|
+
});
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
### Stream Events
|
|
392
|
+
|
|
393
|
+
All compression and decompression streams emit the following events:
|
|
394
|
+
|
|
395
|
+
```typescript
|
|
396
|
+
import { compress } from "simple-zstd";
|
|
397
|
+
|
|
398
|
+
const stream = await compress(3);
|
|
399
|
+
|
|
400
|
+
// Standard Duplex stream events
|
|
401
|
+
stream.on("data", (chunk: Buffer) => {
|
|
402
|
+
console.log("Received chunk:", chunk.length, "bytes");
|
|
403
|
+
});
|
|
404
|
+
|
|
405
|
+
stream.on("end", () => {
|
|
406
|
+
console.log("Stream finished");
|
|
407
|
+
});
|
|
408
|
+
|
|
409
|
+
stream.on("error", (err: Error) => {
|
|
410
|
+
console.error("Stream error:", err);
|
|
411
|
+
});
|
|
412
|
+
|
|
413
|
+
// zstd-specific events
|
|
414
|
+
stream.on("stderr", (message: string) => {
|
|
415
|
+
// zstd process stderr output
|
|
416
|
+
console.warn("zstd stderr:", message);
|
|
417
|
+
});
|
|
418
|
+
|
|
419
|
+
stream.on("exit", (code: number, signal: NodeJS.Signals | null) => {
|
|
420
|
+
// zstd process exit event
|
|
421
|
+
console.log("zstd process exited with code:", code);
|
|
422
|
+
});
|
|
168
423
|
```
|
|
169
424
|
|
|
170
|
-
|
|
425
|
+
**Event Reference:**
|
|
426
|
+
|
|
427
|
+
- `data` - Emitted when compressed/decompressed data is available
|
|
428
|
+
- `end` - Emitted when the stream has finished processing
|
|
429
|
+
- `error` - Emitted on stream errors or if zstd exits with non-zero code
|
|
430
|
+
- `stderr` - Emitted when the zstd process writes to stderr (warnings, debug info)
|
|
431
|
+
- `exit` - Emitted when the underlying zstd process exits
|
|
171
432
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
433
|
+
## Debugging
|
|
434
|
+
|
|
435
|
+
Enable debug output using the `DEBUG` environment variable:
|
|
175
436
|
|
|
176
437
|
```bash
|
|
177
|
-
|
|
438
|
+
# Debug simple-zstd operations
|
|
439
|
+
DEBUG=SimpleZSTD node app.js
|
|
440
|
+
|
|
441
|
+
# Debug process queue
|
|
442
|
+
DEBUG=SimpleZSTDQueue node app.js
|
|
178
443
|
|
|
444
|
+
# Debug both
|
|
445
|
+
DEBUG=SimpleZSTD,SimpleZSTDQueue node app.js
|
|
179
446
|
```
|
|
180
447
|
|
|
448
|
+
## Migrating to v2
|
|
449
|
+
|
|
450
|
+
Version 2.0 is a complete rewrite with TypeScript support and a modernized API. Here's what you need to know:
|
|
451
|
+
|
|
452
|
+
### Breaking Changes
|
|
453
|
+
|
|
454
|
+
#### 1. Node.js Version Requirement
|
|
455
|
+
|
|
456
|
+
**v1:** No explicit requirement
|
|
457
|
+
**v2:** Requires Node.js >= 18.0.0
|
|
458
|
+
|
|
459
|
+
```bash
|
|
460
|
+
# Check your Node version
|
|
461
|
+
node --version # Should be v18.0.0 or higher
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
#### 2. Function Names Changed
|
|
465
|
+
|
|
466
|
+
| v1 Function | v2 Function |
|
|
467
|
+
| ----------------------- | -------------------------------------------- |
|
|
468
|
+
| `ZSTDCompress(level)` | `compress(level, opts?)` |
|
|
469
|
+
| `ZSTDDecompress()` | `decompress(opts?)` |
|
|
470
|
+
| `ZSTDDecompressMaybe()` | `decompress(opts?)` _(built-in auto-detect)_ |
|
|
471
|
+
|
|
472
|
+
**v1 Code:**
|
|
473
|
+
|
|
474
|
+
```javascript
|
|
475
|
+
const { ZSTDCompress, ZSTDDecompress } = require("simple-zstd");
|
|
476
|
+
|
|
477
|
+
const compressStream = ZSTDCompress(3);
|
|
478
|
+
const decompressStream = ZSTDDecompress();
|
|
479
|
+
```
|
|
480
|
+
|
|
481
|
+
**v2 Code:**
|
|
482
|
+
|
|
483
|
+
```javascript
|
|
484
|
+
const { compress, decompress } = require("simple-zstd");
|
|
485
|
+
|
|
486
|
+
const compressStream = await compress(3);
|
|
487
|
+
const decompressStream = await decompress();
|
|
488
|
+
```
|
|
489
|
+
|
|
490
|
+
#### 3. All Functions Now Return Promises
|
|
491
|
+
|
|
492
|
+
**v1:** Functions returned streams synchronously
|
|
493
|
+
**v2:** Functions return `Promise<Duplex>` and must be awaited
|
|
494
|
+
|
|
495
|
+
**v1 Code:**
|
|
496
|
+
|
|
497
|
+
```javascript
|
|
498
|
+
fs.createReadStream("file.txt")
|
|
499
|
+
.pipe(ZSTDCompress(3))
|
|
500
|
+
.pipe(ZSTDDecompress())
|
|
501
|
+
.pipe(fs.createWriteStream("output.txt"));
|
|
502
|
+
```
|
|
503
|
+
|
|
504
|
+
**v2 Code:**
|
|
505
|
+
|
|
506
|
+
```javascript
|
|
507
|
+
const c = await compress(3);
|
|
508
|
+
const d = await decompress();
|
|
509
|
+
|
|
510
|
+
await pipeline(
|
|
511
|
+
fs.createReadStream("file.txt"),
|
|
512
|
+
c,
|
|
513
|
+
d,
|
|
514
|
+
fs.createWriteStream("output.txt")
|
|
515
|
+
);
|
|
516
|
+
```
|
|
517
|
+
|
|
518
|
+
#### 4. "Maybe" Functionality Now Built-in
|
|
519
|
+
|
|
520
|
+
**v1:** Had separate `ZSTDDecompressMaybe()` function
|
|
521
|
+
**v2:** All decompression functions auto-detect and pass through non-compressed data
|
|
522
|
+
|
|
523
|
+
**v1 Code:**
|
|
524
|
+
|
|
525
|
+
```javascript
|
|
526
|
+
const { ZSTDDecompressMaybe } = require("simple-zstd");
|
|
527
|
+
|
|
528
|
+
stream.pipe(ZSTDDecompressMaybe()).pipe(output);
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
**v2 Code:**
|
|
532
|
+
|
|
533
|
+
```javascript
|
|
534
|
+
const { decompress } = require("simple-zstd");
|
|
535
|
+
|
|
536
|
+
const d = await decompress(); // Automatically detects compressed data
|
|
537
|
+
pipeline(stream, d, output);
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
#### 5. Options Structure Changed
|
|
541
|
+
|
|
542
|
+
**v1:** Limited options as separate parameters
|
|
543
|
+
**v2:** Unified options object with TypeScript types
|
|
544
|
+
|
|
545
|
+
**v1 Code:**
|
|
546
|
+
|
|
547
|
+
```javascript
|
|
548
|
+
// v1 had limited customization
|
|
549
|
+
ZSTDCompress(3, streamOptions);
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
**v2 Code:**
|
|
553
|
+
|
|
554
|
+
```javascript
|
|
555
|
+
await compress(3, {
|
|
556
|
+
dictionary: Buffer.from("..."),
|
|
557
|
+
zstdOptions: ["--ultra"],
|
|
558
|
+
spawnOptions: { cwd: "/tmp" },
|
|
559
|
+
streamOptions: { highWaterMark: 64 * 1024 },
|
|
560
|
+
});
|
|
561
|
+
```
|
|
562
|
+
|
|
563
|
+
### New Features in v2
|
|
564
|
+
|
|
565
|
+
#### TypeScript Support
|
|
566
|
+
|
|
567
|
+
```typescript
|
|
568
|
+
import { compress, decompress, SimpleZSTD } from "simple-zstd";
|
|
569
|
+
import type { ZSTDOpts, PoolOpts } from "simple-zstd";
|
|
570
|
+
```
|
|
571
|
+
|
|
572
|
+
#### Buffer Interface
|
|
573
|
+
|
|
574
|
+
New convenience methods for working with buffers directly:
|
|
575
|
+
|
|
576
|
+
```typescript
|
|
577
|
+
import { compressBuffer, decompressBuffer } from "simple-zstd";
|
|
578
|
+
|
|
579
|
+
const compressed = await compressBuffer(Buffer.from("data"), 3);
|
|
580
|
+
const decompressed = await decompressBuffer(compressed);
|
|
581
|
+
```
|
|
582
|
+
|
|
583
|
+
#### Process Pooling with SimpleZSTD Class
|
|
584
|
+
|
|
585
|
+
Pre-spawn processes for better performance with async factory method:
|
|
586
|
+
|
|
587
|
+
```typescript
|
|
588
|
+
import { SimpleZSTD } from "simple-zstd";
|
|
589
|
+
|
|
590
|
+
const zstd = await SimpleZSTD.create({
|
|
591
|
+
compressQueueSize: 2,
|
|
592
|
+
decompressQueueSize: 2,
|
|
593
|
+
compressQueue: {
|
|
594
|
+
compLevel: 3,
|
|
595
|
+
dictionary: Buffer.from("..."), // Optional
|
|
596
|
+
},
|
|
597
|
+
});
|
|
598
|
+
|
|
599
|
+
// Use pooled processes
|
|
600
|
+
const stream = await zstd.compress();
|
|
601
|
+
|
|
602
|
+
// Or override compression level for specific operations
|
|
603
|
+
const stream2 = await zstd.compress(19);
|
|
604
|
+
|
|
605
|
+
// Clean up when done
|
|
606
|
+
zstd.destroy();
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
#### Dictionary Support
|
|
610
|
+
|
|
611
|
+
Full support for compression dictionaries:
|
|
612
|
+
|
|
613
|
+
```typescript
|
|
614
|
+
const dictionary = fs.readFileSync("dict.zstd");
|
|
615
|
+
|
|
616
|
+
await compress(3, { dictionary });
|
|
617
|
+
await compressBuffer(data, 3, { dictionary });
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
### Migration Examples
|
|
621
|
+
|
|
622
|
+
#### Simple Stream Compression
|
|
623
|
+
|
|
624
|
+
**v1:**
|
|
625
|
+
|
|
626
|
+
```javascript
|
|
627
|
+
const { ZSTDCompress, ZSTDDecompress } = require("simple-zstd");
|
|
628
|
+
|
|
629
|
+
fs.createReadStream("input.txt")
|
|
630
|
+
.pipe(ZSTDCompress(3))
|
|
631
|
+
.pipe(fs.createWriteStream("output.zst"));
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
**v2:**
|
|
635
|
+
|
|
636
|
+
```javascript
|
|
637
|
+
const { compress } = require("simple-zstd");
|
|
638
|
+
const { pipeline } = require("node:stream/promises");
|
|
639
|
+
|
|
640
|
+
const c = await compress(3);
|
|
641
|
+
await pipeline(
|
|
642
|
+
fs.createReadStream("input.txt"),
|
|
643
|
+
c,
|
|
644
|
+
fs.createWriteStream("output.zst")
|
|
645
|
+
);
|
|
646
|
+
```
|
|
647
|
+
|
|
648
|
+
#### Error Handling
|
|
649
|
+
|
|
650
|
+
**v1:**
|
|
651
|
+
|
|
652
|
+
```javascript
|
|
653
|
+
ZSTDCompress(3).on("error", (err) => console.error(err));
|
|
654
|
+
```
|
|
655
|
+
|
|
656
|
+
**v2:**
|
|
657
|
+
|
|
658
|
+
```javascript
|
|
659
|
+
try {
|
|
660
|
+
const c = await compress(3);
|
|
661
|
+
c.on("error", (err) => console.error(err));
|
|
662
|
+
} catch (err) {
|
|
663
|
+
console.error("Failed to create stream:", err);
|
|
664
|
+
}
|
|
665
|
+
```
|
|
666
|
+
|
|
667
|
+
#### With Custom Options
|
|
668
|
+
|
|
669
|
+
**v1:**
|
|
670
|
+
|
|
671
|
+
```javascript
|
|
672
|
+
// Limited options in v1
|
|
673
|
+
const stream = ZSTDCompress(3, { highWaterMark: 64 * 1024 });
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
**v2:**
|
|
677
|
+
|
|
678
|
+
```javascript
|
|
679
|
+
const stream = await compress(3, {
|
|
680
|
+
streamOptions: { highWaterMark: 64 * 1024 },
|
|
681
|
+
zstdOptions: ["--ultra"],
|
|
682
|
+
});
|
|
683
|
+
```
|
|
684
|
+
|
|
685
|
+
### Upgrade Checklist
|
|
686
|
+
|
|
687
|
+
- [ ] Update Node.js to >= 18.0.0
|
|
688
|
+
- [ ] Replace `ZSTDCompress` with `compress`
|
|
689
|
+
- [ ] Replace `ZSTDDecompress` with `decompress`
|
|
690
|
+
- [ ] Replace `ZSTDDecompressMaybe` with `decompress` (same function)
|
|
691
|
+
- [ ] Add `await` to all compression/decompression calls
|
|
692
|
+
- [ ] Update imports to use new function names
|
|
693
|
+
- [ ] Consider using buffer methods (`compressBuffer`/`decompressBuffer`) for simpler use cases
|
|
694
|
+
- [ ] Consider using `SimpleZSTD` class for high-throughput scenarios
|
|
695
|
+
- [ ] Update error handling for async/await pattern
|
|
696
|
+
- [ ] Update tests to handle promises
|
|
697
|
+
|
|
698
|
+
## Performance Benchmarks
|
|
699
|
+
|
|
700
|
+
This package has been benchmarked against other zstd packages.
|
|
701
|
+
At this time is appears to be the fastest package for processing large files.
|
|
702
|
+
|
|
703
|
+
[Benchmark Tests](https://github.com/Stieneee/node-compression-test)
|
|
704
|
+
|
|
705
|
+
## Performance Considerations
|
|
706
|
+
|
|
707
|
+
This package spawns a child process for each compression or decompression operation. While this provides excellent performance for large files, child process creation overhead can become a bottleneck when processing many small files rapidly.
|
|
708
|
+
|
|
709
|
+
**Solution:** Use the `SimpleZSTD` class with process pooling for high-throughput scenarios:
|
|
710
|
+
|
|
711
|
+
```typescript
|
|
712
|
+
const zstd = await SimpleZSTD.create({
|
|
713
|
+
compressQueueSize: 4, // Pre-spawn 4 compression processes
|
|
714
|
+
decompressQueueSize: 4, // Pre-spawn 4 decompression processes
|
|
715
|
+
});
|
|
716
|
+
|
|
717
|
+
// Reuse pooled processes for multiple operations
|
|
718
|
+
for (const file of files) {
|
|
719
|
+
const stream = await zstd.compress();
|
|
720
|
+
// ... process file ...
|
|
721
|
+
}
|
|
722
|
+
|
|
723
|
+
zstd.destroy(); // Clean up when done
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
Process pooling significantly reduces latency by reusing existing child processes instead of spawning new ones for each operation.
|
|
727
|
+
|
|
181
728
|
## Contributing
|
|
182
729
|
|
|
183
730
|
Pull requests are welcome.
|
|
@@ -186,7 +733,7 @@ Pull requests are welcome.
|
|
|
186
733
|
|
|
187
734
|
MIT License
|
|
188
735
|
|
|
189
|
-
Copyright (c)
|
|
736
|
+
Copyright (c) 2025 Tyler Stiene
|
|
190
737
|
|
|
191
738
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
192
739
|
of this software and associated documentation files (the "Software"), to deal
|