@medicomind/rolldown-compression 1.0.1 → 1.1.0

package/CHANGELOG.md

@@ -1,5 +1,11 @@
 # @medicomind/rolldown-compression
 
+## 1.1.0
+
+### Minor Changes
+
+- [`1d98d99`](https://github.com/Medico-Mind/rolldown-compression/commit/1d98d995567280c60194451818c7caffb7808bc6) Thanks [@Mnwa](https://github.com/Mnwa)! - Add `stream` option: compress from disk in `writeBundle` (order `post`) instead of in memory in `generateBundle`. Files are read on demand in bounded batches (`chunkSize` source bytes per batch, or 4 MB when `chunkSize` is 0), so the whole build is never buffered by the plugin and assets written to disk by other plugins' `writeBundle` hooks are compressed as well — removing the previous `generateBundle`-only limitation.
+
 ## 1.0.1
 
 ### Patch Changes

package/README.md

@@ -57,6 +57,7 @@ compression({
   skipIfLargerOrEqual: true,
   concurrency: 0, // 0 = number of logical CPUs
   chunkSize: 0, // 0 = compress everything in one batch
+  stream: false, // true = on-demand disk-based compression in writeBundle
   logLevel: 'info',
 })
 ```
@@ -91,7 +92,8 @@ export default defineConfig({
 | `deleteOriginalAssets` | `boolean` | `false` | Remove the original from the bundle after all algorithms processed it. Errors if `filename` resolves to the source name. |
 | `skipIfLargerOrEqual` | `boolean` | `true` | Don't emit artifacts whose compressed size is `>=` the original. |
 | `concurrency` | `number` | `0` | Native worker threads. `0` = number of logical CPUs. |
-| `chunkSize` | `number` | `0` | Max source bytes buffered per native compression batch. `0` = one batch for the whole build. A positive value (e.g. `64 * 1024 * 1024`) caps the plugin's peak memory overhead at roughly one batch of source copies plus its compressed outputs; a single file larger than `chunkSize` still forms its own batch. The bundler keeps the original bundle in memory regardless. |
+| `chunkSize` | `number` | `0` | Max source bytes buffered per native compression batch. `0` = one batch for the whole build. A positive value (e.g. `64 * 1024 * 1024`) caps the plugin's peak memory overhead at roughly one batch of source copies plus its compressed outputs; a single file larger than `chunkSize` still forms its own batch. The bundler keeps the original bundle in memory regardless. In [stream mode](#stream-mode) `0` falls back to a 4 MB batch instead of one big batch. |
+| `stream` | `boolean` | `false` | Compress from disk in `writeBundle` (order `post`) instead of in memory in `generateBundle`: files are read on demand in bounded batches (`chunkSize` bytes, default 4 MB), and assets written to disk by other plugins' `writeBundle` hooks are compressed too (see [stream mode](#stream-mode)). |
 | `logLevel` | `'silent' \| 'error' \| 'warn' \| 'info'` | `'info'` | Plugin log verbosity; `info` prints a per-algorithm summary at build end. |
 | `enableInWatchMode` | `boolean` | `false` | The plugin is a no-op in watch/dev mode unless enabled (see [watch mode](#watch--dev-mode)). |
 
@@ -120,7 +122,23 @@ defineAlgorithm('zstd', { level: 12 })
 - A failing task never aborts the batch: per-task errors are aggregated and fail the build with one message.
 - Already-compressed artifacts (`.gz`, `.br`, `.zst` — ours or pre-existing) are never re-compressed, so chaining plugin instances can't produce `app.js.gz.br`.
 
-Limitation: assets written to disk by other plugins in `writeBundle`/`closeBundle` (i.e. after `generateBundle`) are not seen by this plugin. This matches how `vite-plugin-compression2` handles the in-bundle pass; a `writeBundle` fallback was deliberately left out to keep the pipeline zero-copy and single-pass. File an issue if you have a concrete case.
+Limitation of the default mode: assets written to disk by other plugins in `writeBundle`/`closeBundle` (i.e. after `generateBundle`) are not seen. This matches how `vite-plugin-compression2` handles the in-bundle pass. Set `stream: true` to remove it.
+
+### Stream mode
+
+With `stream: true` the plugin skips the in-memory `generateBundle` pass entirely and instead runs at the end of `writeBundle` (hook order `post`), after the bundle — and any extra assets other plugins wrote in their own `writeBundle` hooks — is on disk:
+
+```ts
+compression({
+  stream: true,
+  chunkSize: 64 * 1024 * 1024, // optional: batch by source bytes instead of by file count
+})
+```
+
+- The output directory is scanned recursively and matching files are read **on demand**, never all at once: a batch is flushed to the native module whenever it reaches `chunkSize` source bytes, defaulting to **4 MB** when `chunkSize` is `0`. Peak memory overhead is one batch of sources plus its compressed outputs, regardless of build size.
+- Compressed artifacts are written straight to the output directory (`emitFile` is not available after write); `deleteOriginalAssets` unlinks the originals from disk.
+- The same filter, threshold and re-compression guards apply, and everything already in the output directory that matches them is compressed — including files produced by other plugins after `generateBundle`, the default mode's limitation. Assets written in `closeBundle` (after all `writeBundle` hooks) are still not seen.
+- Trade-off: files the bundler already had in memory are re-read from disk, and per-batch FFI calls replace the single big batch — for small builds the default in-memory mode is faster.
 
 ### Watch / dev mode
 
@@ -254,7 +272,7 @@ the host, so on an unsupported platform the wasm package must be opted into:
 - **No tarball plugin**: out of scope.
 - **gzip default level is 6** (zlib default), not 9 — measurably faster for a ~1% size difference. Pass `defineAlgorithm('gzip', { level: 9 })` to match the reference.
 - **zstd everywhere**: zstd is compiled in, with no dependency on the Node runtime's zstd support (node >= 22.15).
-- Extra options: `concurrency` (native thread cap), `chunkSize` (memory cap per compression batch) and `enableInWatchMode`.
+- Extra options: `concurrency` (native thread cap), `chunkSize` (memory cap per compression batch), `stream` (on-demand disk-based compression in `writeBundle`) and `enableInWatchMode`.
 
 ## Implementation decisions
 

package/dist/index.cjs

@@ -25,6 +25,7 @@ let node_crypto = require("node:crypto");
 let node_path = require("node:path");
 node_path = __toESM(node_path);
 let _rollup_pluginutils = require("@rollup/pluginutils");
+let node_fs_promises = require("node:fs/promises");
 let node_module = require("node:module");
 //#region ts/options.ts
 /**
@@ -127,7 +128,7 @@ function validateDefined(entry) {
 /** Validate and normalize user options. Throws at `compression()` call time. */
 function resolveOptions(options = {}) {
 	if (options === null || typeof options !== "object") throw new OptionValidationError("options must be an object");
-	const { include = DEFAULT_INCLUDE, exclude, threshold = 0, algorithms = ["gzip", "brotli"], filename, deleteOriginalAssets = false, skipIfLargerOrEqual = true, concurrency = 0, chunkSize = 0, logLevel = "info", enableInWatchMode = false } = options;
+	const { include = DEFAULT_INCLUDE, exclude, threshold = 0, algorithms = ["gzip", "brotli"], filename, deleteOriginalAssets = false, skipIfLargerOrEqual = true, concurrency = 0, chunkSize = 0, stream = false, logLevel = "info", enableInWatchMode = false } = options;
 	if (typeof threshold !== "number" || Number.isNaN(threshold) || threshold < 0) throw new OptionValidationError(`invalid threshold: expected a non-negative number, got ${JSON.stringify(threshold)}`);
 	if (!Number.isInteger(concurrency) || concurrency < 0) throw new OptionValidationError(`invalid concurrency: expected a non-negative integer, got ${JSON.stringify(concurrency)}`);
 	if (!Number.isInteger(chunkSize) || chunkSize < 0) throw new OptionValidationError(`invalid chunkSize: expected a non-negative integer number of bytes, got ${JSON.stringify(chunkSize)}`);
@@ -143,6 +144,7 @@ function resolveOptions(options = {}) {
 		skipIfLargerOrEqual: Boolean(skipIfLargerOrEqual),
 		concurrency,
 		chunkSize,
+		stream: Boolean(stream),
 		logLevel,
 		enableInWatchMode: Boolean(enableInWatchMode)
 	};
@@ -167,7 +169,28 @@ function resolveOutputFileName(filename, fileName, algorithm, source) {
 const compressBuffers = (0, node_module.createRequire)(require("url").pathToFileURL(__filename).href)("../index.js").compressBuffers;
 //#endregion
 //#region ts/plugin.ts
+/**
+* Rolldown plugin implementation.
+*
+* Default mode: all eligible assets of a build are collected in
+* `generateBundle` (while they are still in memory) and compressed in
+* batched FFI calls; the native module fans the work out across a rayon
+* thread pool without ever blocking the JS event loop. By default
+* everything goes out in a single batch; a positive `chunkSize` flushes a
+* batch whenever its source bytes reach that limit, so only one batch of
+* buffer copies is alive at a time.
+*
+* Stream mode (`stream: true`): compression instead runs at the end of
+* `writeBundle` (order `'post'`), scanning the output directory on disk.
+* Files are read on demand and processed in bounded batches — `chunkSize`
+* source bytes per batch, falling back to
+* {@link STREAM_DEFAULT_CHUNK_SIZE} when `chunkSize` is 0 — so the whole
+* build is never held in memory and assets written to disk by other
+* plugins' `writeBundle` hooks are compressed as well.
+*/
 const PLUGIN_NAME = "rolldown-compression";
+/** Source bytes per batch in stream mode when `chunkSize` is 0. */
+const STREAM_DEFAULT_CHUNK_SIZE = 4 * 1024 * 1024;
 const LOG_PRIORITY = {
 	silent: 0,
 	error: 1,
@@ -197,97 +220,175 @@ function formatBytes(bytes) {
 	if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} kB`;
 	return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
 }
+/**
+* Shared batching core: queues per-algorithm tasks, flushes them to the
+* native module and hands successful results to `emit`. Only one batch of
+* source buffers is referenced at a time once a flush trigger is set.
+*/
+function createBatchRunner(options, logger, emittedNames, emit, fail) {
+	const processedSources = /* @__PURE__ */ new Set();
+	const emittedBySource = /* @__PURE__ */ new Map();
+	const failures = [];
+	const stats = /* @__PURE__ */ new Map();
+	const chunkSize = options.chunkSize > 0 ? options.chunkSize : options.stream ? STREAM_DEFAULT_CHUNK_SIZE : 0;
+	let pending = [];
+	let pendingSourceBytes = 0;
+	const flush = async () => {
+		if (pending.length === 0) return;
+		const batch = pending;
+		pending = [];
+		pendingSourceBytes = 0;
+		const results = await compressBuffers(batch.map((artifact) => artifact.task), batch.map((artifact) => artifact.buffer), {
+			concurrency: options.concurrency,
+			skipIfLargerOrEqual: options.skipIfLargerOrEqual
+		});
+		for (const [index, result] of results.entries()) {
+			const artifact = batch[index];
+			if (artifact === void 0) continue;
+			if (result.error !== void 0 && result.error !== null) {
+				failures.push(`${result.fileName} (${result.algorithm}): ${result.error}`);
+				continue;
+			}
+			if (result.skipped) {
+				logger.info(`skipped ${artifact.outputFileName}: ${result.algorithm} output would not be smaller than the original`);
+				continue;
+			}
+			await emit(artifact, result.data);
+			emittedNames.add(artifact.outputFileName);
+			emittedBySource.set(artifact.sourceFileName, (emittedBySource.get(artifact.sourceFileName) ?? 0) + 1);
+			const stat = stats.get(result.algorithm) ?? {
+				count: 0,
+				originalBytes: 0,
+				outputBytes: 0
+			};
+			stat.count += 1;
+			stat.originalBytes += result.originalSize;
+			stat.outputBytes += result.compressedSize;
+			stats.set(result.algorithm, stat);
+		}
+	};
+	const addFile = async (fileName, buffer) => {
+		for (const algorithm of options.algorithms) {
+			const outputFileName = resolveOutputFileName(options.filename, fileName, algorithm, buffer);
+			if (outputFileName === fileName) fail(`the filename option resolved "${fileName}" (${algorithm.algorithm}) to the same name as the source asset; refusing to overwrite it`);
+			pending.push({
+				task: {
+					fileName,
+					algorithm: algorithm.algorithm,
+					level: algorithm.level,
+					windowBits: algorithm.windowBits
+				},
+				buffer,
+				sourceFileName: fileName,
+				outputFileName
+			});
+		}
+		processedSources.add(fileName);
+		pendingSourceBytes += buffer.byteLength;
+		if (chunkSize > 0 && pendingSourceBytes >= chunkSize) await flush();
+	};
+	return {
+		addFile,
+		flush,
+		processedSources,
+		emittedBySource,
+		failures,
+		stats
+	};
+}
+function logSummary(logger, stats, startedAt) {
+	const summary = [...stats.entries()].map(([algorithm, stat]) => `${algorithm}: ${stat.count} file(s), ${formatBytes(stat.originalBytes)} -> ${formatBytes(stat.outputBytes)}`).join("; ");
+	if (summary.length === 0) return;
+	const totalSaved = [...stats.values()].reduce((sum, stat) => sum + (stat.originalBytes - stat.outputBytes), 0);
+	const elapsedMs = performance.now() - startedAt;
+	logger.info(`${summary}; saved ${formatBytes(totalSaved)} in ${(elapsedMs / 1e3).toFixed(2)}s`);
+}
+/** Recursively list every file under `root`, in a deterministic order. */
+async function walkFiles(root) {
+	const files = [];
+	const visit = async (dir) => {
+		const entries = await (0, node_fs_promises.readdir)(dir, { withFileTypes: true });
+		entries.sort((a, b) => a.name < b.name ? -1 : a.name > b.name ? 1 : 0);
+		for (const entry of entries) {
+			const absolute = node_path.default.join(dir, entry.name);
+			if (entry.isDirectory()) await visit(absolute);
+			else if (entry.isFile()) files.push(absolute);
+		}
+	};
+	await visit(root);
+	return files;
+}
 function createCompressionPlugin(options) {
 	const logger = createLogger(options.logLevel);
 	const emittedNames = /* @__PURE__ */ new Set();
 	return Object.assign({
 		name: PLUGIN_NAME,
 		async generateBundle(_outputOptions, bundle) {
+			if (options.stream) return;
 			if (this.meta?.watchMode && !options.enableInWatchMode) {
 				logger.info("watch mode detected, skipping compression (set enableInWatchMode to opt in)");
 				return;
 			}
 			const startedAt = performance.now();
-			const processedSources = /* @__PURE__ */ new Set();
-			const failures = [];
-			const emittedBySource = /* @__PURE__ */ new Map();
-			const stats = /* @__PURE__ */ new Map();
-			let pending = [];
-			let pendingSourceBytes = 0;
-			const flush = async () => {
-				if (pending.length === 0) return;
-				const batch = pending;
-				pending = [];
-				pendingSourceBytes = 0;
-				const results = await compressBuffers(batch.map((artifact) => artifact.task), batch.map((artifact) => artifact.buffer), {
-					concurrency: options.concurrency,
-					skipIfLargerOrEqual: options.skipIfLargerOrEqual
+			const fail = (message) => this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${message}`));
+			const runner = createBatchRunner(options, logger, emittedNames, (artifact, data) => {
+				this.emitFile({
+					type: "asset",
+					fileName: artifact.outputFileName,
+					source: data
 				});
-				for (const [index, result] of results.entries()) {
-					const artifact = batch[index];
-					if (artifact === void 0) continue;
-					if (result.error !== void 0 && result.error !== null) {
-						failures.push(`${result.fileName} (${result.algorithm}): ${result.error}`);
-						continue;
-					}
-					if (result.skipped) {
-						logger.info(`skipped ${artifact.outputFileName}: ${result.algorithm} output would not be smaller than the original`);
-						continue;
-					}
-					this.emitFile({
-						type: "asset",
-						fileName: artifact.outputFileName,
-						source: result.data
-					});
-					emittedNames.add(artifact.outputFileName);
-					emittedBySource.set(artifact.sourceFileName, (emittedBySource.get(artifact.sourceFileName) ?? 0) + 1);
-					const stat = stats.get(result.algorithm) ?? {
-						count: 0,
-						originalBytes: 0,
-						outputBytes: 0
-					};
-					stat.count += 1;
-					stat.originalBytes += result.originalSize;
-					stat.outputBytes += result.compressedSize;
-					stats.set(result.algorithm, stat);
-				}
-			};
+			}, fail);
 			for (const [fileName, output] of Object.entries(bundle)) {
 				if (!options.filter(fileName)) continue;
 				if (COMPRESSED_EXTENSION_RE.test(fileName) || emittedNames.has(fileName)) continue;
 				const buffer = toBuffer(output.type === "chunk" ? output.code : output.source);
 				if (buffer.byteLength < options.threshold) continue;
-				for (const algorithm of options.algorithms) {
-					const outputFileName = resolveOutputFileName(options.filename, fileName, algorithm, buffer);
-					if (outputFileName === fileName) this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] the filename option resolved "${fileName}" (${algorithm.algorithm}) to the same name as the source asset; refusing to overwrite it`));
-					pending.push({
-						task: {
-							fileName,
-							algorithm: algorithm.algorithm,
-							level: algorithm.level,
-							windowBits: algorithm.windowBits
-						},
-						buffer,
-						sourceFileName: fileName,
-						outputFileName
-					});
-				}
-				processedSources.add(fileName);
-				pendingSourceBytes += buffer.byteLength;
-				if (options.chunkSize > 0 && pendingSourceBytes >= options.chunkSize) await flush();
+				await runner.addFile(fileName, buffer);
 			}
-			await flush();
-			if (processedSources.size === 0) return;
-			if (failures.length > 0) this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${failures.length} compression task(s) failed:\n${failures.join("\n")}`));
-			if (options.deleteOriginalAssets) for (const fileName of processedSources) {
-				if ((emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
+			await runner.flush();
+			if (runner.processedSources.size === 0) return;
+			if (runner.failures.length > 0) fail(`${runner.failures.length} compression task(s) failed:\n${runner.failures.join("\n")}`);
+			if (options.deleteOriginalAssets) for (const fileName of runner.processedSources) {
+				if ((runner.emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
 				delete bundle[fileName];
 			}
-			const elapsedMs = performance.now() - startedAt;
-			const summary = [...stats.entries()].map(([algorithm, stat]) => `${algorithm}: ${stat.count} file(s), ${formatBytes(stat.originalBytes)} -> ${formatBytes(stat.outputBytes)}`).join("; ");
-			if (summary.length > 0) {
-				const totalSaved = [...stats.values()].reduce((sum, stat) => sum + (stat.originalBytes - stat.outputBytes), 0);
-				logger.info(`${summary}; saved ${formatBytes(totalSaved)} in ${(elapsedMs / 1e3).toFixed(2)}s`);
+			logSummary(logger, runner.stats, startedAt);
+		},
+		writeBundle: {
+			order: "post",
+			async handler(outputOptions) {
+				if (!options.stream) return;
+				if (this.meta?.watchMode && !options.enableInWatchMode) {
+					logger.info("watch mode detected, skipping compression (set enableInWatchMode to opt in)");
+					return;
+				}
+				const outDir = outputOptions.dir !== void 0 ? node_path.default.resolve(outputOptions.dir) : outputOptions.file !== void 0 ? node_path.default.resolve(node_path.default.dirname(outputOptions.file)) : void 0;
+				if (outDir === void 0) {
+					logger.warn("stream mode could not determine the output directory, skipping compression");
+					return;
+				}
+				const startedAt = performance.now();
+				const fail = (message) => this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${message}`));
+				const runner = createBatchRunner(options, logger, emittedNames, async (artifact, data) => {
+					const target = node_path.default.join(outDir, artifact.outputFileName);
+					await (0, node_fs_promises.mkdir)(node_path.default.dirname(target), { recursive: true });
+					await (0, node_fs_promises.writeFile)(target, data);
+				}, fail);
+				for (const absolute of await walkFiles(outDir)) {
+					const fileName = node_path.default.relative(outDir, absolute).split(node_path.default.sep).join("/");
+					if (!options.filter(fileName)) continue;
+					if (COMPRESSED_EXTENSION_RE.test(fileName) || emittedNames.has(fileName)) continue;
+					if ((await (0, node_fs_promises.stat)(absolute)).size < options.threshold) continue;
+					await runner.addFile(fileName, await (0, node_fs_promises.readFile)(absolute));
+				}
+				await runner.flush();
+				if (runner.processedSources.size === 0) return;
+				if (runner.failures.length > 0) fail(`${runner.failures.length} compression task(s) failed:\n${runner.failures.join("\n")}`);
+				if (options.deleteOriginalAssets) for (const fileName of runner.processedSources) {
+					if ((runner.emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
+					await (0, node_fs_promises.unlink)(node_path.default.join(outDir, fileName));
+				}
+				logSummary(logger, runner.stats, startedAt);
 			}
 		}
 	}, { apply: "build" });

package/dist/index.d.cts

@@ -79,6 +79,16 @@ interface CompressionOptions {
    * @default 0
    */
   chunkSize?: number;
+  /**
+   * Compress from disk at the end of `writeBundle` instead of in memory in
+   * `generateBundle`. The output directory is scanned and files are read on
+   * demand in bounded batches (`chunkSize` source bytes per batch, or 4 MB
+   * per batch when `chunkSize` is 0), so the whole build is never held in
+   * memory and assets written to disk by other plugins' `writeBundle`
+   * hooks are picked up as well.
+   * @default false
+   */
+  stream?: boolean;
   /** @default 'info' */
   logLevel?: LogLevel;
   /**

package/dist/index.d.mts

@@ -79,6 +79,16 @@ interface CompressionOptions {
    * @default 0
    */
   chunkSize?: number;
+  /**
+   * Compress from disk at the end of `writeBundle` instead of in memory in
+   * `generateBundle`. The output directory is scanned and files are read on
+   * demand in bounded batches (`chunkSize` source bytes per batch, or 4 MB
+   * per batch when `chunkSize` is 0), so the whole build is never held in
+   * memory and assets written to disk by other plugins' `writeBundle`
+   * hooks are picked up as well.
+   * @default false
+   */
+  stream?: boolean;
   /** @default 'info' */
   logLevel?: LogLevel;
   /**

package/dist/index.mjs

@@ -2,6 +2,7 @@ import { createRequire } from "node:module";
 import { createHash } from "node:crypto";
 import path from "node:path";
 import { createFilter } from "@rollup/pluginutils";
+import { mkdir, readFile, readdir, stat, unlink, writeFile } from "node:fs/promises";
 //#region ts/options.ts
 /**
 * Option types, normalization and validation for the public API.
@@ -103,7 +104,7 @@ function validateDefined(entry) {
 /** Validate and normalize user options. Throws at `compression()` call time. */
 function resolveOptions(options = {}) {
 	if (options === null || typeof options !== "object") throw new OptionValidationError("options must be an object");
-	const { include = DEFAULT_INCLUDE, exclude, threshold = 0, algorithms = ["gzip", "brotli"], filename, deleteOriginalAssets = false, skipIfLargerOrEqual = true, concurrency = 0, chunkSize = 0, logLevel = "info", enableInWatchMode = false } = options;
+	const { include = DEFAULT_INCLUDE, exclude, threshold = 0, algorithms = ["gzip", "brotli"], filename, deleteOriginalAssets = false, skipIfLargerOrEqual = true, concurrency = 0, chunkSize = 0, stream = false, logLevel = "info", enableInWatchMode = false } = options;
 	if (typeof threshold !== "number" || Number.isNaN(threshold) || threshold < 0) throw new OptionValidationError(`invalid threshold: expected a non-negative number, got ${JSON.stringify(threshold)}`);
 	if (!Number.isInteger(concurrency) || concurrency < 0) throw new OptionValidationError(`invalid concurrency: expected a non-negative integer, got ${JSON.stringify(concurrency)}`);
 	if (!Number.isInteger(chunkSize) || chunkSize < 0) throw new OptionValidationError(`invalid chunkSize: expected a non-negative integer number of bytes, got ${JSON.stringify(chunkSize)}`);
@@ -119,6 +120,7 @@ function resolveOptions(options = {}) {
 		skipIfLargerOrEqual: Boolean(skipIfLargerOrEqual),
 		concurrency,
 		chunkSize,
+		stream: Boolean(stream),
 		logLevel,
 		enableInWatchMode: Boolean(enableInWatchMode)
 	};
@@ -143,7 +145,28 @@ function resolveOutputFileName(filename, fileName, algorithm, source) {
 const compressBuffers = createRequire(import.meta.url)("../index.js").compressBuffers;
 //#endregion
 //#region ts/plugin.ts
+/**
+* Rolldown plugin implementation.
+*
+* Default mode: all eligible assets of a build are collected in
+* `generateBundle` (while they are still in memory) and compressed in
+* batched FFI calls; the native module fans the work out across a rayon
+* thread pool without ever blocking the JS event loop. By default
+* everything goes out in a single batch; a positive `chunkSize` flushes a
+* batch whenever its source bytes reach that limit, so only one batch of
+* buffer copies is alive at a time.
+*
+* Stream mode (`stream: true`): compression instead runs at the end of
+* `writeBundle` (order `'post'`), scanning the output directory on disk.
+* Files are read on demand and processed in bounded batches — `chunkSize`
+* source bytes per batch, falling back to
+* {@link STREAM_DEFAULT_CHUNK_SIZE} when `chunkSize` is 0 — so the whole
+* build is never held in memory and assets written to disk by other
+* plugins' `writeBundle` hooks are compressed as well.
+*/
 const PLUGIN_NAME = "rolldown-compression";
+/** Source bytes per batch in stream mode when `chunkSize` is 0. */
+const STREAM_DEFAULT_CHUNK_SIZE = 4 * 1024 * 1024;
 const LOG_PRIORITY = {
 	silent: 0,
 	error: 1,
@@ -173,97 +196,175 @@ function formatBytes(bytes) {
 	if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} kB`;
 	return `${(bytes / (1024 * 1024)).toFixed(2)} MB`;
 }
+/**
+* Shared batching core: queues per-algorithm tasks, flushes them to the
+* native module and hands successful results to `emit`. Only one batch of
+* source buffers is referenced at a time once a flush trigger is set.
+*/
+function createBatchRunner(options, logger, emittedNames, emit, fail) {
+	const processedSources = /* @__PURE__ */ new Set();
+	const emittedBySource = /* @__PURE__ */ new Map();
+	const failures = [];
+	const stats = /* @__PURE__ */ new Map();
+	const chunkSize = options.chunkSize > 0 ? options.chunkSize : options.stream ? STREAM_DEFAULT_CHUNK_SIZE : 0;
+	let pending = [];
+	let pendingSourceBytes = 0;
+	const flush = async () => {
+		if (pending.length === 0) return;
+		const batch = pending;
+		pending = [];
+		pendingSourceBytes = 0;
+		const results = await compressBuffers(batch.map((artifact) => artifact.task), batch.map((artifact) => artifact.buffer), {
+			concurrency: options.concurrency,
+			skipIfLargerOrEqual: options.skipIfLargerOrEqual
+		});
+		for (const [index, result] of results.entries()) {
+			const artifact = batch[index];
+			if (artifact === void 0) continue;
+			if (result.error !== void 0 && result.error !== null) {
+				failures.push(`${result.fileName} (${result.algorithm}): ${result.error}`);
+				continue;
+			}
+			if (result.skipped) {
+				logger.info(`skipped ${artifact.outputFileName}: ${result.algorithm} output would not be smaller than the original`);
+				continue;
+			}
+			await emit(artifact, result.data);
+			emittedNames.add(artifact.outputFileName);
+			emittedBySource.set(artifact.sourceFileName, (emittedBySource.get(artifact.sourceFileName) ?? 0) + 1);
+			const stat = stats.get(result.algorithm) ?? {
+				count: 0,
+				originalBytes: 0,
+				outputBytes: 0
+			};
+			stat.count += 1;
+			stat.originalBytes += result.originalSize;
+			stat.outputBytes += result.compressedSize;
+			stats.set(result.algorithm, stat);
+		}
+	};
+	const addFile = async (fileName, buffer) => {
+		for (const algorithm of options.algorithms) {
+			const outputFileName = resolveOutputFileName(options.filename, fileName, algorithm, buffer);
+			if (outputFileName === fileName) fail(`the filename option resolved "${fileName}" (${algorithm.algorithm}) to the same name as the source asset; refusing to overwrite it`);
+			pending.push({
+				task: {
+					fileName,
+					algorithm: algorithm.algorithm,
+					level: algorithm.level,
+					windowBits: algorithm.windowBits
+				},
+				buffer,
+				sourceFileName: fileName,
+				outputFileName
+			});
+		}
+		processedSources.add(fileName);
+		pendingSourceBytes += buffer.byteLength;
+		if (chunkSize > 0 && pendingSourceBytes >= chunkSize) await flush();
+	};
+	return {
+		addFile,
+		flush,
+		processedSources,
+		emittedBySource,
+		failures,
+		stats
+	};
+}
+function logSummary(logger, stats, startedAt) {
+	const summary = [...stats.entries()].map(([algorithm, stat]) => `${algorithm}: ${stat.count} file(s), ${formatBytes(stat.originalBytes)} -> ${formatBytes(stat.outputBytes)}`).join("; ");
+	if (summary.length === 0) return;
+	const totalSaved = [...stats.values()].reduce((sum, stat) => sum + (stat.originalBytes - stat.outputBytes), 0);
+	const elapsedMs = performance.now() - startedAt;
+	logger.info(`${summary}; saved ${formatBytes(totalSaved)} in ${(elapsedMs / 1e3).toFixed(2)}s`);
+}
+/** Recursively list every file under `root`, in a deterministic order. */
+async function walkFiles(root) {
+	const files = [];
+	const visit = async (dir) => {
+		const entries = await readdir(dir, { withFileTypes: true });
+		entries.sort((a, b) => a.name < b.name ? -1 : a.name > b.name ? 1 : 0);
+		for (const entry of entries) {
+			const absolute = path.join(dir, entry.name);
+			if (entry.isDirectory()) await visit(absolute);
+			else if (entry.isFile()) files.push(absolute);
+		}
+	};
+	await visit(root);
+	return files;
+}
 function createCompressionPlugin(options) {
 	const logger = createLogger(options.logLevel);
 	const emittedNames = /* @__PURE__ */ new Set();
 	return Object.assign({
 		name: PLUGIN_NAME,
 		async generateBundle(_outputOptions, bundle) {
+			if (options.stream) return;
 			if (this.meta?.watchMode && !options.enableInWatchMode) {
 				logger.info("watch mode detected, skipping compression (set enableInWatchMode to opt in)");
 				return;
 			}
 			const startedAt = performance.now();
-			const processedSources = /* @__PURE__ */ new Set();
-			const failures = [];
-			const emittedBySource = /* @__PURE__ */ new Map();
-			const stats = /* @__PURE__ */ new Map();
-			let pending = [];
-			let pendingSourceBytes = 0;
-			const flush = async () => {
-				if (pending.length === 0) return;
-				const batch = pending;
-				pending = [];
-				pendingSourceBytes = 0;
-				const results = await compressBuffers(batch.map((artifact) => artifact.task), batch.map((artifact) => artifact.buffer), {
-					concurrency: options.concurrency,
-					skipIfLargerOrEqual: options.skipIfLargerOrEqual
+			const fail = (message) => this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${message}`));
+			const runner = createBatchRunner(options, logger, emittedNames, (artifact, data) => {
+				this.emitFile({
+					type: "asset",
+					fileName: artifact.outputFileName,
+					source: data
 				});
-				for (const [index, result] of results.entries()) {
-					const artifact = batch[index];
-					if (artifact === void 0) continue;
-					if (result.error !== void 0 && result.error !== null) {
-						failures.push(`${result.fileName} (${result.algorithm}): ${result.error}`);
-						continue;
-					}
-					if (result.skipped) {
-						logger.info(`skipped ${artifact.outputFileName}: ${result.algorithm} output would not be smaller than the original`);
-						continue;
-					}
-					this.emitFile({
-						type: "asset",
-						fileName: artifact.outputFileName,
-						source: result.data
-					});
-					emittedNames.add(artifact.outputFileName);
-					emittedBySource.set(artifact.sourceFileName, (emittedBySource.get(artifact.sourceFileName) ?? 0) + 1);
-					const stat = stats.get(result.algorithm) ?? {
-						count: 0,
-						originalBytes: 0,
-						outputBytes: 0
-					};
-					stat.count += 1;
-					stat.originalBytes += result.originalSize;
-					stat.outputBytes += result.compressedSize;
-					stats.set(result.algorithm, stat);
-				}
-			};
+			}, fail);
 			for (const [fileName, output] of Object.entries(bundle)) {
 				if (!options.filter(fileName)) continue;
 				if (COMPRESSED_EXTENSION_RE.test(fileName) || emittedNames.has(fileName)) continue;
 				const buffer = toBuffer(output.type === "chunk" ? output.code : output.source);
 				if (buffer.byteLength < options.threshold) continue;
-				for (const algorithm of options.algorithms) {
-					const outputFileName = resolveOutputFileName(options.filename, fileName, algorithm, buffer);
-					if (outputFileName === fileName) this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] the filename option resolved "${fileName}" (${algorithm.algorithm}) to the same name as the source asset; refusing to overwrite it`));
-					pending.push({
-						task: {
-							fileName,
-							algorithm: algorithm.algorithm,
-							level: algorithm.level,
-							windowBits: algorithm.windowBits
-						},
-						buffer,
-						sourceFileName: fileName,
-						outputFileName
-					});
-				}
-				processedSources.add(fileName);
-				pendingSourceBytes += buffer.byteLength;
-				if (options.chunkSize > 0 && pendingSourceBytes >= options.chunkSize) await flush();
+				await runner.addFile(fileName, buffer);
 			}
-			await flush();
-			if (processedSources.size === 0) return;
-			if (failures.length > 0) this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${failures.length} compression task(s) failed:\n${failures.join("\n")}`));
-			if (options.deleteOriginalAssets) for (const fileName of processedSources) {
-				if ((emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
+			await runner.flush();
+			if (runner.processedSources.size === 0) return;
+			if (runner.failures.length > 0) fail(`${runner.failures.length} compression task(s) failed:\n${runner.failures.join("\n")}`);
+			if (options.deleteOriginalAssets) for (const fileName of runner.processedSources) {
+				if ((runner.emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
 				delete bundle[fileName];
 			}
-			const elapsedMs = performance.now() - startedAt;
-			const summary = [...stats.entries()].map(([algorithm, stat]) => `${algorithm}: ${stat.count} file(s), ${formatBytes(stat.originalBytes)} -> ${formatBytes(stat.outputBytes)}`).join("; ");
-			if (summary.length > 0) {
-				const totalSaved = [...stats.values()].reduce((sum, stat) => sum + (stat.originalBytes - stat.outputBytes), 0);
-				logger.info(`${summary}; saved ${formatBytes(totalSaved)} in ${(elapsedMs / 1e3).toFixed(2)}s`);
+			logSummary(logger, runner.stats, startedAt);
+		},
+		writeBundle: {
+			order: "post",
+			async handler(outputOptions) {
+				if (!options.stream) return;
+				if (this.meta?.watchMode && !options.enableInWatchMode) {
+					logger.info("watch mode detected, skipping compression (set enableInWatchMode to opt in)");
+					return;
+				}
+				const outDir = outputOptions.dir !== void 0 ? path.resolve(outputOptions.dir) : outputOptions.file !== void 0 ? path.resolve(path.dirname(outputOptions.file)) : void 0;
+				if (outDir === void 0) {
+					logger.warn("stream mode could not determine the output directory, skipping compression");
+					return;
+				}
+				const startedAt = performance.now();
+				const fail = (message) => this.error(/* @__PURE__ */ new Error(`[${PLUGIN_NAME}] ${message}`));
+				const runner = createBatchRunner(options, logger, emittedNames, async (artifact, data) => {
+					const target = path.join(outDir, artifact.outputFileName);
+					await mkdir(path.dirname(target), { recursive: true });
+					await writeFile(target, data);
+				}, fail);
+				for (const absolute of await walkFiles(outDir)) {
+					const fileName = path.relative(outDir, absolute).split(path.sep).join("/");
+					if (!options.filter(fileName)) continue;
+					if (COMPRESSED_EXTENSION_RE.test(fileName) || emittedNames.has(fileName)) continue;
+					if ((await stat(absolute)).size < options.threshold) continue;
+					await runner.addFile(fileName, await readFile(absolute));
+				}
+				await runner.flush();
+				if (runner.processedSources.size === 0) return;
+				if (runner.failures.length > 0) fail(`${runner.failures.length} compression task(s) failed:\n${runner.failures.join("\n")}`);
+				if (options.deleteOriginalAssets) for (const fileName of runner.processedSources) {
+					if ((runner.emittedBySource.get(fileName) ?? 0) === 0) logger.warn(`deleteOriginalAssets removed "${fileName}" even though no compressed variant was emitted for it`);
+					await unlink(path.join(outDir, fileName));
+				}
+				logSummary(logger, runner.stats, startedAt);
 			}
 		}
 	}, { apply: "build" });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@medicomind/rolldown-compression",
-  "version": "1.0.1",
+  "version": "1.1.0",
   "description": "Fast native (Rust) gzip / brotli / zstd compression plugin for Rolldown, powered by napi-rs",
   "keywords": [
     "rolldown",
@@ -91,13 +91,13 @@
     "rolldown": "^1.0.0"
   },
   "optionalDependencies": {
-    "@medicomind/rolldown-compression-darwin-arm64": "1.0.1",
-    "@medicomind/rolldown-compression-darwin-x64": "1.0.1",
-    "@medicomind/rolldown-compression-linux-arm64-gnu": "1.0.1",
-    "@medicomind/rolldown-compression-linux-x64-gnu": "1.0.1",
-    "@medicomind/rolldown-compression-linux-x64-musl": "1.0.1",
-    "@medicomind/rolldown-compression-wasm32-wasi": "1.0.1",
-    "@medicomind/rolldown-compression-win32-x64-msvc": "1.0.1"
+    "@medicomind/rolldown-compression-darwin-arm64": "1.1.0",
+    "@medicomind/rolldown-compression-darwin-x64": "1.1.0",
+    "@medicomind/rolldown-compression-linux-arm64-gnu": "1.1.0",
+    "@medicomind/rolldown-compression-linux-x64-gnu": "1.1.0",
+    "@medicomind/rolldown-compression-linux-x64-musl": "1.1.0",
+    "@medicomind/rolldown-compression-wasm32-wasi": "1.1.0",
+    "@medicomind/rolldown-compression-win32-x64-msvc": "1.1.0"
   },
   "devDependencies": {
     "@biomejs/biome": "^2.5.3",